Electronics maker Pioneer Corp hit a nine-year low after it said it would post a record loss in the current business year, while Shin-Etsu Chemical Co Ltd gained after its earnings beat expectations.
Koichiro Suzuki, a senior investment manager at Sompo Japan Asset Management, said slackening interest from foreign investors could be behind the Nikkei's recent slump.
"If you look at the pre-opening orders from foreign brokerage houses, the frequency of net selling is starting to stand out. I think it's likely that foreigners who want to take profits from the recent rise are now in the majority," he said.
Buying by foreign investors has helped drive the Nikkei to four-year highs this year.
In August, foreigners bought a whopping 1.96 trillion yen ($16.92 billion) in Japanese stocks - the largest net buying in a single month since the Tokyo Stock exchange began compiling such data in 1982 - but their buying has not kept up that break-neck pace.
In September, for instance, they bought a net 1.52 trillion in Japanese stocks.
The Nikkei finished the day down 93.77 points at 13,106.18.
It has fallen 10 times in the 13 sessions since it rose to a four-year closing high of 13,738.84 on October 4.
The broader TOPIX index lost 0.64 percent or 8.87 points to 1,376.50. Trade volume fell, with 2.02 billion shares changing hands on the Tokyo exchange's first section, the lowest level since September 8.
Decliners outnumbered gainers 1,060 to 503. Computers-to-chips conglomerate NEC fell 4.9 percent to 581 yen. Financial newspaper Nihon Keizai reported at the weekend that NEC would likely post a fall in operating profit for the business year to next March, rather than its forecast gain, due to sluggish microchip operations.
Electronics maker Pioneer lost 3.3 percent to 1,515 yen after falling to 1,513 yen, its lowest since 1995.
On Friday, the company said it expected to post its biggest loss on record this business year, reversing its previous forecast of a small profit.
Katsuhiko Kodama, head of equities at Toyo Securities, said a desire to see earnings results was one of the factors keeping investors from buying.
"High-tech stocks are going to see a lot of movement depending on their earnings," he said.
Shin-Etsu, the world's largest maker of silicon wafers, rose 2 percent to 5,090 yen.
After the midsession break, the company posted a 19.7 percent rise in net profit for the April-September first-half and revised up its full-year forecast to a level that beat market expectations.
END TO THE BARGAIN SALE Sompo Japan's Suzuki said foreigners had been drawn to Japanese stocks because shares were undervalued compared with other markets. But he said Japanese stocks are no longer such a bargain.
Analyst Stefan Rheinwald of CLSA Asia-Pacific Markets agreed.
In a strategy note on Monday, Rheinwald told investors that that Japanese stocks are "no longer cheap by international standards".
"We remain positive on Japanese equities, but believe the current rerating (revaluation) phase is over. Without new catalysts, the upside is unlikely to exceed single digits," he said.
Machinery makers fell after US firm Caterpillar Inc, the world's biggest maker of earth-moving equipment, posted disappointing results and warned that higher raw material costs and production bottlenecks would crimp profits.
Komatsu Ltd, Japan's largest maker of construction equipment, lost 3.4 percent to 1,411 yen. Kubota Corp, Japan's largest manufacturer of farm equipment, fell 3.8 percent to 781 yen.
Hitachi Construction Machinery Co Ltd, a manufacturer of hydraulic shovels, fell 4.6 percent to 1,947 yen. Gainers included Isuzu Motors Ltd, which rose 2.7 percent to 451 yen.
The Nihon Keizai said on Monday that General Motors Corp planned to turn over its commercial truck businesses in Australia and South Africa to Isuzu as the struggling US auto giant focuses on its passenger car operations.